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.74 Introduction 


Probability of detection (POD) analysis is used in assessing reliably detectable flaw size in 
nondestructive evaluation (NDE). 


MIL-HDBK- 1823! and associated mh1823* POD software gives most common methods of POD 
analysis. 


Real flaws such as cracks and crack-like flaws are desired to be detected using these NDE 
methods. 


A reliably detectable crack size is required for safe life analysis of fracture critical parts. 


The paper provides discussion on optimizing probability of detection (POD) demonstration 
experiments using Point Estimate Method. POD Point estimate method is used by NASA for 
qualifying special NDE procedures. The point estimate method uses binomial distribution for 
probability density. Normally, a set of 29 flaws of same size within some tolerance are used in the 
demonstration. 


The optimization is performed to provide acceptable value for probability of passing demonstration 
(PPD) and achieving acceptable value for probability of false (POF) calls while keeping the flaw 
sizes in the set as small as possible. 
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Overview 


¢ Traditionally largest flaw size in the set is considered to be a conservative estimate of the flaw size with 
minimum 90% probability and 95% confidence. 
¢ The Point Estimate flaw size is denoted as a 


90 95PE- 
/ 


¢ The paper investigates relationship between range of flaw sizes in relation to Ag9, 1.e. 90% probability flaw 
size, to provide a desired PPD. 


¢ The range of flaw sizes is expressed as a proportion of the standard deviation of the probability density 
distribution. 


¢ Difference between median or average of the 29 flaws and dogp is also expressed as a proportion of standard 
deviation of the probability density distribution. 


¢ In general, it is concluded that, if probability of detection increases with flaw size, average of 29 flaw sizes 
would always be larger than or equal to —_ and is an acceptable measure of a, 


Ago 


¢ If NDE technique has sufficient sensitivity and signal-to-noise ratio, 


0,95PE 
/ 


¢ then the 29 flaw-set can be optimized to meet requirements of minimum required PPD, 

¢ maximum allowable POF, 

* requirements on flaw size tolerance about mean flaw size and 

° flaw size detectability requirements. 

¢ The paper provides procedure for optimizing flaw sizes in the point estimate demonstration flaw-set. 
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a Background, Two Types of POD Datasets 


¢ MIL-HDBK-1823! and associated mh18237 software cover two types of datasets. 


¢ First type of dataset is signal response 4 (read as a-hat) versus flaw size “a’’. 


¢ The dG (y-axis) versus “a” (x-axis) data may be transformed using logarithm function along appropriate 
axes, if needed, to create linear correlation around the decision threshold, Gj, -;c:on- 


¢ A generalized linear model (GLM) is fitted to the transformed data for analysis. 
¢ Here, noise data is taken separately to define noise distribution. 
¢ Noise is same as signal response from part where there is no flaw. 
¢ Noise data is used to determine false call rate or probability of false calls (POF). 
¢ Second type of dataset is called hit-miss data, which contains flaw size and corresponding detection result 1.e. 
hit or miss. 
¢ Hit has numerical value of 1 and miss has numerical value of 0. 


¢ Here, false call data is noted to determine false call rate using Clopper-Pearson binomial distribution 
function. 


¢ Normally, POD increases with flaw size and POF decreases with flaw size. 
¢ POF value shall be within certain limit to prevent adverse impact on cost and schedule. 


¢ ASTM E 28623 also provides the hit-miss POD data analysis method that is consistent with MIL-HDBK- 
1823. 
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Binomial Point Estimation of POD 


¢ Binomial point estimation of POD requires detection of all 29 flaws of same size out of 29 flaw detection 
opportunities, i.e. 29/29. 


¢ Point estimation method is described by Rummel!®. Another variation is detecting 45 flaws out of 46 flaw 
detection opportunities. We will only work on the 29/29 case here. 


¢ These, hit-miss versus detection opportunity numbers come from the cumulative binomial distribution, which 
provides confidence value for given number of hits, opportunities and probability. 


¢ This function is also given in Microsoft Excel. is calculated in Excel as follows, 
¢ BINOMDIST (28, 29, 0.9, TRUE) = 0.9529. For 29 hits in 29 opportunities for POD of 0.9, a confidence of 0.95 
¢ BINOMDIST (28, 29, 0.97637, TRUE) = 0.5 For 29 hits in 29 opportunities for POD of 0.9763, a confidence of 0.5 


¢ ASTM E 2862 gives the Clopper-Pearson binomial method for constructing confidence intervals for 
proportions. It can be used for POD (lower bound) and POF (upper bound) confidence level e.g. 50%, 90% 
and 95%. The Clopper-Pearson upper 100(1-a) % confidence bound for POF, p is, 
-1 
n-X 
Pu = {2 or aR . (1) 
* where x is misses, n is opportunities, a is confidence level, and FU—a, 2x+2, 2n—2x) is the F-statistics with degrees of 
freedom (2x+2, 2n—2x) and P[F < F(1—a, 2x+2, 2n—2x)] = 1—-a. This method is consistent with that used in MIL- 
HDBK-1823. 
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a Optimizing Point Estimate Method 


¢ Usually, hardest-to-detect flaw configuration is used in the 29 flaw-set to envelope other less severe flaw 
detection cases. 


¢ Point estimate method assures that A. ene > 290 and normally the margin is assumed to be very large. 
/ 


° But as a. ..,, comes closer to true Agg, it becomes difficult to pass the demonstration 1.e. Probability of 
passing démonstration (PPD) becomes lower. Moreover, the false call rate may become unacceptable. 


¢ Since, true Aoo is mostly a characteristic of an NDE technique, the point estimate demonstration flaw 
sizes should take into account Qo 9 for the technique if optimal Q oepe is desired. 
/ 


¢ Following requirements are needed to be met in optimizing a point estimate flaw-set. 
¢ 1. Probability of passing demonstration (PPD) test shall be more than a set value (i.e. > 50%). 
¢ 2. Probability of false (POF) calls shall be lower than a set value (i.e. 2%). 


¢ 3. Flaw size range shall be practical to achieve as flaw manufacturing process has inherent tolerance on flaw 


SIZe. 
° 4. cere shall be less than or equal to the design requirement flaw size (e.g. safe life initial flaw size). 
¢ 5. Optimal flaw size is based on following rules. If above requirements can be met, then smallest a flaw 


; : : F 5 
size 1s desirable. If only some of the above requirements can be met, then choose smallest Qh cpp flaw size 


that meets some of the chosen requirements e.g. design requirement, PPD or POF. ee 
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wc Approach 


¢ This approach is based on hypothesis that @ versus “a” curve-fit POD, @ versus “a” mh1823 POD, hit-miss 
mh1823 POD approaches provide lower dg9/95 flaw sizes compared to point estimate POD. 


¢ Therefore, if these methods are used to determinate POD curves, perform noise analysis, choose decision 
threshold, and perform POF analysis, then this information can be used to optimize the point estimate 29 
flaw-set. 


¢ We would first check the hypothesis that average flaw size of the 29 flaws in the point estimate set provides 
— that is greater than true oq of the data model and dg9/95 of POD analysis models which assume that 
POD increases with flaw size. 


¢ We would use mh1823 G@ versus “a” analysis or hit-miss analysis to generate the POD curve. POD curves for 
the method are determined using a flaw-set per MIL-HDBK-1823. 


¢ For @ versus “a” analysis, decision threshold level (DTL) is needed. This level is based on achieving desired 
flaw detectability size but also on limiting POF as calculated using noise data and flaw response data. 


¢ Noise data is taken and DTL versus POF curve is determined. 


¢ Once, the DTL level is determined to provide acceptable POF, it is used to complete the POD analysis and PO 
curve equation is determined. 
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Approach (continued) 


¢ Similarly, for hit-miss analysis, if the NDE technique does not use a DTL, then data on false calls is obtained 
and analyzed per MIL-HDBK-1823 using Clopper-Pearson method. 


¢ Here, we may reduce the POF by increasing size of a relevant indication which also then results in larger 
90/95 - 
* If the POF value is acceptable and d99/95 1s acceptable, then we can proceed to design the point estimate set. 


¢ Here we need to recognize that the point estimate flaw size is larger than true d99 by certain margin denoted 
by A (delta). 


Arequirement = 290/95PE = 299 + A (2) 
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Case 1: Flaw-set with Uniformly Distributed Sizes with Average Size of do, 


¢ Objective in Case | is to study effect of flaw range on PPD. 
¢ The flaw-set has uniformly distributed flaws with median and average flaw size equal to ago. 


¢ We would use a POD model in the study. Standard deviation of a uniformly spaced flaw-set is 30.4% of the 
range. 


¢ It is assumed that signal response @ relates to flaw size “a” as follows. 
a=f,at+fPpot 6 (3) 
¢ Although a linear relationship is chosen, other relationships as given in MIL-HDBK-1823 also apply. 


¢ First, a symmetrical POD function curve based on error function (erf) is chosen. This is given by cumulative 
density distribution of a probability density function which is chosen to be a normal distribution. 


¢ This meets the key assumption that POD increases with flaw size. Probability density function (PDF) in the 
form of normal distribution is given by, 


@ 20% | (4) 


f@) === 
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Model POD Function 


¢ POD function is given by cumulative density distribution (CDF) of the normal distribution PDF. It is given 
by, 


g(a, 4,0) = 5 [1+ erf (FF) (5) 


¢ Where, is mean and a is standard deviation of the PDF and CDF functions. 90% POD is given by following 
expression, 0.9 = g (1.2815, 0, 1). Following CDF expression from Matlab is used. 


g(a) = 0.5[1 + erf(C,a — C,)] (6) 
C, = 1/oVv2, and (7) 
C, = w/ov2. (8) 


¢ We choose C, = 30.8; C, = 7.8 to generate the POD function. 
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Assumed POD Function 


¢ We calculate the following flaw sizes in arbitrary units. 
© Atg=0.1, a,) = 0.2239. 
° At g=0.9, Agg = 0.283. 


e Standard deviation can be calculated as, 


0 = (Agog — A409) /(2 X 1.285) ~ 0.023 (9) 
¢ From Eq. 7, we can calculate the standard deviation in POD model as, 
CH= 20). (10) 
¢ From Eq. 8, mean used is calculated as, 
u = Crav2. (11) 
¢ Here, we construct a set of 29 flaw sizes that are uniformly spaced with median and mean at doo. Range 
R is given by, 
R= Qmax — &min (12) 
° where, d,,,x and d,,;, = are the maximum and minimum flaw sizes in the set of 29 flaws. 
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<a Range and Standard Deviation of Flaw Sizes as % of POD Standard Deviation 


Case 1 condition: a set of 29 flaw sizes that are uniformly spaced with median and mean at Ao9 


¢ Range is given as percentage of standard deviation by following equation, 


Ry, = 100R/c. (13) 
¢ We can also express the standard deviation of flaw sizes as percentage of standard deviation by following 
equation. 
Srlaw%y = 30R/o. (14) 


¢ This allows us to investigate effect of range as percentage of standard deviation of POD function and 
therefore conclusions would be generically applicable, if POD functions are similar. We can simply multiply 
the range by 30% and get the corresponding standard deviation in the flaw sizes. Range and a,,,, and a,,., 
relate to the median size as follows, 


Amin, Umax = Aedian + R/2. (15) 
¢ Spacing between flaws and the range are related by, 
S=R/28. (16) 
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Average of PODs of 29 Flaws 


Average Probability of 29/29 Demonstration 


Case 1 Condition: a set of 29 flaw sizes that are uniformly spaced with median and mean at Qo9 
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Next we calculate average of PODs of the 29 flaws. 
Plot of average probabilities is shown in Fig. 1. 

At a range equal to standard deviation, the average 
probability 0.89. 

The average POD in above chart is either 0.9 or lower. 
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Probability of Passing the Demonstration 


Case 1 Condition: a set of 29 flaw sizes that are uniformly spaced with median and mean at Qo9 


Probability of Passing the 29 Demonstration 
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Fig. 2: Probability of passing the demonstration 


We also calculate product of PODs of 29 flaws which is same as the 
probability of passing (PPD) the 29/29 demonstration. 

The POD needs to be paired with its corresponding PPD for proper 
interpretation. Probability of passing the demonstration is given in Fig. 2. 

The above chart indicates that as range increases, it becomes harder to pass the 
29/29 demonstration or PPD gets lower. 

Since, goal is to pass the demonstration, it is wise to have shortest range in 
flaw sizes. 

At a range equal to one standard deviation, PPD is 0.035 with confidence of 
95%. 

This is also equivalent to PPD of 0.5 with confidence of 0.5. 

This PPD is too low for establishing successful demonstration protocol. 

We would only use 95% confidence in rest of the paper. At range equal to two 
standard deviations, PPD decreases from 0.05 to 0.012. 

Therefore, although there is no requirement on range of the flaws, if average 
flaw size is close to true Agq then flaw range would be very small too. 

Here we compute PPD versus PODs for single size flaw 29/29 demonstration 
by using following equation, 


PPD, =1—a (18) 


SPIE Smart Structures/NDE 2017, Portland, OR 17 


Fig. 


Probability of Passing Demonstration for Flaw Sizes 


4S _in a Range and for Single Size 


Case 1 Condition: a set of 29 flaw sizes that are uniformly spaced with median and mean at Qo9 
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3: Probability of passing 29/29 demonstration for 
flaw sizes in a range and for single flaw size 


Here, binomial distribution is used to compute confidence 
for the same average POD values. 

PPD versus POD for single size flaws is compared with the 
same for flaws in a range. 


For lower values of average POD, the range of flaw sizes is 
larger. See also Fig. 1. 

Fig. 3 indicates that for same average flaw size, PPD with 
flaws in a range is less than or equal to PPD for single size 
flaws. 

Therefore, average flaw size of a set of 29 flaws can be used 
as a more conservative a... estimate compared to single size 
are estimate, provided POD i increases with flaw size. 

If we increase the median flaw size to be greater than the true 
Aog, then it should be easier to pass the demonstration at the 
cost of increased a... from the demonstration. 

This point is discussed in Case 2. 
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Case 2: Median Flaw Size = dogo 


Case 2 condition: Probability of passing ¢ In Case 2, our goal is to study relationship between the flaw 

demonstration equals 5% size range of a uniformly distributed 29 flaw-set and the 

median (or average) flaw size in relation to Agog based on 
assumed POD curve to obtain a PPD of 0.05. 

¢ We would continue with the same POD curve as in Case 1. 

Here, we define delta as difference between median size of 


o 
co 
~™ 


o 
R 


o 
a 


the flaw-set and true or model input agp as follows, 


° 
= 


(19) 
0.03 A= Amedian — %90 
¢ We express delta as percentage of standard deviation by, 
oNe Ay = 100 (Ameaian — 490)/¢ - (20) 


Probability of Passing 29/29 Demonstration 


° We choose flaw spacing or range as % of standard deviation 
of the POD curve and construct a uniformly distributed set of 
0 oF 1 15 2 25 3 35 4 29 flaws about a median flaw size providing probability of 
pele n Peeve passing the demonstration ~ 0.05. 
Fig. 4: Calculated PPD versus % spacing when optimal °* Fig. 4 shows a number of % spacing values for which 
delta was calculated. corresponding % delta values providing PPD of 0.05 were 
determined. See Fig. 4. 
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Case 2: Median Flaw Size = dogo 


Case 2 condition: Probability of passing demonstration equals 5% 
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The % delta versus % range providing PPD 
of 0.05 are provided in Fig. 5. 


As expected, the average POD would 
increase with flaw spacing when PPD is held 
constant. 


In order to satisfy a condition that PPD of 
uniformly spaced 29 flaws is same as PPD of 
single sized 29 flaws with POD of 0.9, % 
delta and % range get related. % delta 
increases as % range increases. In this study, 
% delta and % spacing are approximately 
related by following equation. 


20 
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Fig. 8: Average POD versus% flaw 
spacing for 0.5 PPD 


This curve also has some small error due to 

discrete values used in iterative 

calculations. 

¢ The curve starts at 0.97637 POD for 
single size flaws. 

¢ Average POD seems to level out at 
0.983 for flaw spacing of ~250% for 
PPD = 0.5. Fig. 9 shows % delta versus 
% flaw spacing. 


Case 2: Median Flaw Size = dogo 


Case 2 condition: Probability of passing demonstration equals 50% 
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Fig. 9: Average POD versus % flaw spacing and delta versus range 
¢ We determine Qo for the same decision threshold level using MIL-HDBK-1823. 
Fig. 9 indicates that, in order to get PPD value of 50%, a % delta of 69.5% 
(~70%) is necessary for single size flaw-set. 
¢ In general, 


Aso% = Ago/95PE — A990 = 9.6950. (24) 


¢ For flaw spacing of 7% of POD model standard deviation and % delta with 0.5 
PPD is about standard deviation. 


Aso% = A90/95PE — A990 =O. (25) 
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<a Decision Threshold for 50% Probability of Passing Demonstration 


Case 2 condition: Probability of passing demonstration equals 50% 


¢ For flaw spacing of 7%, corresponding standard deviation of flaw-set is, 0 7 x 28 x 0.304 = 59.6%. 


¢ This indicates that the point estimate demonstration that is reasonably easy to pass 1s likely to be 
very conservative. 


¢ It also implies that the decision threshold level (DTL for a-hat versus “a”’ testing) signals or the 
relevant flaw size (hit-miss testing) which establishes dg, is lower than average signal from flaw 
size a by about | standard deviation in signal. Signal response delta 1s given by, 
ae 


A(Aso%) = A(Aoo/o5pz) — A(Aoo) = (0). (26) 
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‘.cz4 Signal to Noise Ratio 


¢ DTL signal-to-noise ratio should be high (1.e. > 2) to limit POF. 
* Here, signal margin fora, |_| flaw size above DTL is same as standard deviation for noise times As 0. 
/ 


* Noise G,,,,,. can be measured as signal in areas where no flaws are present. mh1823 calculates percentile noise 
(e.g. cumulative 95% or G,,,;.95,) and gives a curve for POF as a function of percentile noise. Therefore, the 
DTL-to-noise ratio can be calculated as, 

SNRoriy% = 4pri/Gnoises9%- (27) 

¢ If we do not have actual noise measurements, we can still use standard deviation of the POD curve as a 
measure of noise. 

SNRoris = Gpri/A(o). (28) 
¢ Even this signal-to-noise ratio should be high (> 2). If both these SNR are high it provides a low POF. 


¢ In hit-miss analysis, signal response may not be measured but here the ratio with relevant flaw size may be of 
use. Relevant flaw size is due to signal or indication from the smallest flaw that needs to be reported. 
Therefore, it is a kind of DTL in flaw size units. Following flaw size ratio (FSR) would affect POF. Higher 
ratio would imply lower POF. Possibly, a minimum ratio above certain value (e.g. 1) would be desirable. 


FSRreio = Are /0 (29) 


¢ Following flaw size ratio (FSR) would affect POD. Higher ratio would imply higher POD. Possibly, a 
minimum ratio above certain value (e.g. 1) would be desirable. 
FSRoo/o5PE = (99 /95PE — Are) /o (30) 
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hee Steps for Optimizing Point Estimate POD 


¢ Step 1. Choose PPD value for the demonstration, e.g. 50%. 
¢ Step 2. Choose decision threshold or relevant flaw size. 
¢ Step 3. Collect hit-miss data or @ versus “a” data for the flaw detection application. 


¢ Step 4. Determine POD curve using MIL-HDBK-1823. Determine dog (a, wo)? mean (a. ~o) and standard 
deviation in flaw units. : : 


¢ Step 5. Choose a delta above 70% and determine % flaw spacing. 


¢ Step 6. Construct flaw-set as follows. 


Amedian = 499 + %Asoy%o/100, (31) 
S= %S59%,0/100, and (32) 
Amin) Imax = Imedian + 14S. (33) 


¢ One could calculate standard deviation of the flaw-set from S$ and use mean as median to design the flaw-set 
for non-uniformly spaced flaws. This would give an approximate answer. 


¢ Once flaws are chosen, then the PPD can be directly calculated by multiplying the PODs for 29 flaws and 
adjustment to the flaw-set made to provide acceptable PPD, POF and average POD. 


¢ This leads us to Case 3, where we would compare several a... of a simulated data with simulation input doo 
value. 
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Case 3: Comparison of Various d99/95 Estimates from Simulated Data 


Table 1: Comparison of various d99/95 estimates from simulated data 
Simulated data with programmed do, = 0.2984 


Method Quantity | Value % Difference 
(Arbitrary with Model 
4.55 c c c c c cE E c x units) Ago 
—/=— Pit 
4- ~80% Prediction Bounds 
a Theoretical aa 0.2984 Not applicable 
3.5/- (based on 
< simulation 
<= 
6 38> model) 
2 mh1823 cone 0.2907 -2.6 
je} 
Q 2.5/- dversus “a” 
3g Ago /95 0.2946 Allee: 1 
S 2b mh1823- Ag0/s0 0.2977 -0.2 
od) 
Ss leah 
. ibisnlss ee 0.3119 4.5 3 
5) 
Matlab curve fit — dg9/s0 0.2895 -3.0 
al v4 | versus “a ae 0.2989 0.2 ) 
oe : | Point estimate gg /ospe 0.323 8.2 : 
01 o2 08 O04 O05 O06 O07 O08 089 1 based on 
Flaw Size, a average of 29 
Fig. 10: Simulated signal response versus flaw size data contiguous 
flaws 
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Fig. 11: Distribution function fit to noise data 


Fit: Maximum Likelihood Estimation (MLE), 


Distribution: Normal, 

Log likelihood: -143.012, 
Domain: -Infinity < y < Infinity, 
Mean, mu: -0.110793, 

Variance: 1.0329, 

Sigma: 1.01631. 
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Fig. 12: a) Cumulative noise data and cumulative probability density 


function, b) Cumulative probability density function or POD 


and 


noise 


confidence bounds. 


Probability of false calls (POF) is given by POD 


equal to the decision threshold. 


POF = 1 - POD noise 


noisee Where noise level is 


(34) 


Refer to MIL-HDBK-1823 for in depth analysis of noise and estimation 


of probability of false calls. 
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-e® Conclusions 


Case 1 indicates that for same average flaw size, PPD with flaws in a range is less than or equal to 
PPD for single size flaws. 
¢ Therefore, average flaw size of a set of 29 flaws can be used as a more conservative a, 
estimate compared to single size a... estimate, provided POD increases with flaw size! 
a 


07> 


Case 2 indicates that in order to get PPD of 50%, a % delta of 69.5% is necessary for single size 29 
flaw set. 


For flaw spacing of 7% of POD model standard deviation and PPD of 50%, % delta is equal 
tol00% of standard deviation. 


For flaw spacing of 7% of POD model standard deviation, the standard deviation of the flaw-set is 
59.6% of POD model standard deviation. 


In Case 3, POD analysis on simulated data indicates that, point estimate a. op flaw size is the 


most conservative among the a, __ from four POD analysis methods discussed. 
¢ PPD was calculated to be 53%. 
¢ Delta is calculated to be 107%. 
¢ Flaw spacing is calculated 4.35% using the same standard deviation. 
¢ Flaw-set standard deviation is 37% of the POD model standard deviation. 
¢ The DTL SNR were calculated to be much greater than 2. 
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7 Conclusions (continued) 


In Case 3, Simulated data analysis results agree with Case 2 predictions to some extent. 


“a 99 


Compared to @ versus method, point estimate method has a lower DTL to qualify the same 


90,95 
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Lowering the DTL would detect smaller flaws reliably but also may cause higher false call rate. 
The POD input data model and mh1823 @ versus “a” model differ and therefore the Case 3 results 
do not come very close to the Case 2 predictions but they are of same order. 


The paper indicates that for PPD of 50%, point estimate a — 1S very conservative possibly by as 
much as the standard deviation of the POD model with standard deviation for 29 flaw-set equal to 
40-60% of standard deviation of the POD model. 


In order to design point estimate 29 flaw-set, the paper recommends using mh1823 POD analysis 
first. 


Results of the mh1823 POD analysis e.g. POD curve, noise percentile, POF and POD versus DTL 
analysis shall be used to design the optimal point estimate flaw-set that provides smallest possible 
with acceptable PPD and POF. 
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Procedure for optimizing the point estimate demonstration set is provided. 
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